Auto-encoders: reconstruction versus compression

We discuss the similarities and differences between training an autoencoder to minimize the reconstruction error, and training the same auto-encoder to compress the data via a generative model. Given a dataset, auto-encoders (for instance, [PH87, Section 8.1] or [HS06]) aim at building a hopefully simpler representation of the data via a hidden, usually lower-dimensional feature space. This is done by looking for a pair of maps X f → Y g → X from data space X to feature space Y and back, such that the reconstruction error between x and g(f(x)) is small. Identifying relevant features hopefully makes the data more understandable, more compact, or simpler to describe. Here we take this interpretation literally, by considering auto-encoders in the framework of minimum description length, i.e., data compression via a probabilistic generative model, using the general correspondence between compression and “simple” probability distributions on the data (for instance, [Grü07]). The objective is then to minimize the codelength (log-likelihood) of the data using the features found by the auto-encoder1. This note answers two questions. First, do auto-encoders trained to minimize reconstruction error actually minimize the length of a compressed encoding of the data, or an approximation thereof? Second, if the goal is to compress the data, why should the auto-encoder approach help compared to directly looking for a generative model? We will see that by adding an information-theoretic term to the reconstruction error, auto-encoders can be trained to minimize a tight upper bound on the codelength (compressed size) of the data. In most situations, directly working with the codelength is difficult (Section 2) hence the interest of such bounds. In Section 3 we introduce a first, simple bound on codelength based on reconstruction error; it is not tight and is valid only for discrete features. Still it already illustrates how minimizing codelength favors using fewer features. The goal is not to explicitly build an encoding of the data [Grü07]; rather, it is to find a good pair of feature and generative functions that would yield a short codelength. If the codelength is known as a function of the parameters of the auto-encoder, it can be used as the training criterion.